IMAGE SELECTING APPARATUS , CAMERA, AND METHOD OF SELECTING IMAGE 

This patent application claims priority based on Japanese 
patent application, Hll-158668 filed on June 04, 1999, the 
contents of which are incorporated herein by reference. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to an image selecting 
apparatus and a camera, and more particularly to an image 
selecting apparatus and a camera capable of automatically 
selecting a desired image from among a plurality of photographed 
images . 

2 . Description of the Related Art 

Conventionally, a technique is known to correct a 
photograph so that a person photographed by a camera can be 
satisfied with the result. However, this technique requires a 
high degree of skill. Furthermore, it is difficult to correct 
a person's face in the photograph when he or she is blinking or 
is not smiling, to a face as if he or she is not blinking or is 
smiling . 

On the other hand, Japanese Patent Laid-open Publication 
(Kokai) H9— 212620 and Japanese Patent Laid-open Publication 
(Kokai) H10— 191216 disclose a technique to continuously 
photograph a plurality of images. Those images are displayed, 
and the person photographed by the camera can select a desirable 
image from among those images. 

However, this was troublesome because the photographed 
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person or the photographer needed to select the desired image by 
checking all of the images. Furthermore, when a lot of people 
are photographed in the image, it is more difficult to select an 
image that all of them are satisfied' with. 

5 

SUMMARY OF THE INVENTION 

Therefore, it is an object of the present invention to 
provide an image selecting apparatus, a camera, and a method of 
10 selecting an image, which overcomes the above issues in the 

related art. This object is achieved by combinations described 
in the independent claims. The dependent claims define further 
advantageous and exemplary combinations of the present 
invention. 

01 15 

%1 In order to solve the above- stated problem, the first 

^ aspect of the present invention provides an image selecting 

p apparatus for selecting a desired image from among a plurality 

JJJ of images obtained by continuously photographing a subject, 

Rj 20 comprising: an extractor extracting data of an aimed object from 
n each of the plurality of images; a condition-storing unit storing 

a predetermined selection condition for a desirable aimed object; 

and a selecting unit selecting a desired image including a desired 

aimed object from among the plurality of images, the desired aimed 
25 object satisfying the predetermined selection condition stored 

in the condition- storing unit. 

The extractor may extract the data of the aimed object based 
on depth information indicating the distance to each part of the 
30 subj ect . 

The extractor may extract the data of the aimed object based 
on image information included in each of the images. 



FF-0006 



The extractor may detect a judgement location from the data 
of the aimed object based on image information included in each 
of the images, the selection condition may include a 
predetermined selection condition related to a desirable 
5 judgement location, and the selecting unit may select the desired 
aimed object including a judgement location satisfying the 
selection condition related to the desirable judgement location. 

The extractor may extract data of a plurality of the aimed 
10 objects from each of the plurality of images; and the selecting 
unit may select a plurality of the desired aimed objects for each 
of the plurality of aimed objects. 

JfJ The extractor may detect a plurality of judgement locations 

yfl 15 from each of the data of the plurality of aimed objects based on 
image information included in each of the images, the selection 
s ~~ condition may include a predetermined selection condition 

is 

Q related to a desirable judgement location, and the selecting unit 

on 

^ may select the plurality of the desired aimed objects each 

[U 20 including a judgement location satisfying the selection 

S condition related to the desirable judgement location. 

The selecting unit may further comprise an image-composite 
unit compositing the plurality of desired aimed objects to form 
25 a composite image, the composite image including the plurality 
of desired aimed objects for each of the plurality of aimed 
objects extracted from the plurality of images. 

In order to solve the above-stated problem, the second 
30 aspect of the present invention provides a camera comprising: an 
input unit forming an image of a subject; an extractor extracting 
data of an aimed object from each of the plurality of images formed 
by the input unit; a condition-storing unit storing a 
predetermined selection condition for a desirable aimed object; 
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and a selecting unit selecting c. desired image including a desired 
aimed object from among the plurality of images, the desired aimed 
object satisfying the predetermined selection condition stored 
in the condition-storing unit. 

The input unit may include a parallactic image data input 
unit inputting parallactic an image photographed from different 
view points, and the extractor may extract the data of the aimed 
object based on depth information indicating the distance to each 
part of the subject, the depth information being extracted from 
the parallactic image. 

The selection condition may comprise a plurality of 
selection conditions, and the camera may further comprise a 
condition-setting unit previously selecting at least one of the 
selection conditions, for selecting the desired image, from among 
the plurality of selection conditions. 

In order to solve the above-stated problem, the third 
aspect of the present invention provides a method of selecting 
a desired image from among a plurality of images obtained by 
continuously photographing a subject, comprising: extracting 
data of an aimed object from each of the plurality of images; and 
selecting a desired image including a desired aimed object from 
among the plurality of images, the desired aimed object 
satisfying a predetermined selection condition for a desirable 
aimed object. 

The extracting may extract the data of the aimed object from 
each of the plurality of images based on depth information 
indicating the distance to each part of the subject. 

The extracting may extract the data of the aimed object from 
each of the plurality of images based on image information 
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included in each of the images. 

The extracting may include detecting a judgement location 
from the data of the aimed object, the selection condition may 
include a predetermined selection condition related to a 
desirable judgement location, and the selecting may select the 
desired aimed object including a judgement location satisfying 
the selection condition related to the desirable judgement 
location . 

The extracting may extract data of a plurality of the aimed 
objects from each of the plurality of images; and the selecting 
may select a plurality of the desired aimed objects for each of 
the plurality of aimed objects. 



The extracting may include detecting a plurality of 
judgement locations from each of the data of the plurality of 
aimed objects, the selection condition may include a 
predetermined selection condition related to a desirable 
[U 20 judgement location, and the selecting may select the plurality 
p of the desired aimed objects each including a judgement location 

satisfying the selection condition related to the desirable 
j udgement locat ion . 

25 In order to solve the above -stated problem, the fourth 

aspect of the present invention provides a recording medium 
storing therein a program executed by a computer to perform a 
method of selecting a desired image from among a plurality of 
images obtained by continuously photographing a subject, 

30 comprising: extracting data of an aimed object from each of the 
plurality of images; and selecting a desired image including a 
desired aimed object from among the plurality of images, the 
desired aimed object satisfying a predetermined selection 
condition for a desirable aimed object. 
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This summary of the invention does not necessarily describe 
all necessary features so that the invention may also be a 
sub- combination of these described features. 

5 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 shows a camera 10 of the first embodiment according 
to the present invention, 

Fig. 2 is a block diagram of the control unit 50 according 
to the present embodiment, 

Fig. 3 is a functional block diagram of the extractor 60, 
Fig. 4 is a functional block diagram of the selecting unit 

80, 

Fig. 5 is a flowchart showing the method of selecting a 
desired image from among a plurality of images photographed by 
the camera 10, 

Fig. 6 is a flowchart showing in detail the method of 
extracting a face part, step 106 in Fig. 5, 

Fig. 7 is a flowchart showing in detail the method of 
detecting a judgement location, step 108 in Fig. 5, 

Fig. 8 is a flowchart showing in detail the method of 
selecting a desired image, step 112 in Fig. 5, and 

Fig. 9 shows a Laboratory system of the second embodiment 
according to the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

The invention will now be described based on the preferred 
30 embodiments, which do not intend to limit the scope of the present 
invention, but exemplify the invention. All of the features and 
the combinations thereof described in the embodiment are not 
necessarily essential to the invention. 
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Fig. 1 shows a camera 10 of the first embodiment according 
to the present invention. The camera 10 continuously photographs 
a plurality of images and automatically selects a desired image 
from among the plurality of images based on a predetermined 
5 selection condition. 

The camera 10 includes an input unit 20, an A/D converter 
30, a memory 40, a control unit 50, a recording unit 90 and an 
output unit 92 . The camera 10 may be, for example, a digital still 
10 camera or a digital video camera that can photograph a still 
image . 

;S The input unit 2 0 includes a parallactic image data input 

m unit 22 and a normal image data input unit 24. The parallactic 

15 image data input unit 22 inputs parallactic images which are 
Q photographed from different view points. The parallactic image 

s data input unit 22 has a parallactic lens 32, a parallactic 

shutter 34, and a parallactic charge coupled device (CCD) 36. The 

yl 

G parallactic lens 32 forms an image of a subject. The parallactic 

m 20 shutter 34 has a plurality of shutter units each of which serves 
O as view points. The parallactic shutter 34 opens one of the 

plurality of shutter units. The parallactic CCD 36 receives the 
image of the subject through the parallactic lens 32 and whichever 
of the shutter units of the parallactic shutter 34 that are opened 
25 The parallactic CCD 36 also receives another image of the subject 
through the parallactic lens 32 and another of the shutter units 
of the parallactic shutter 34, which is opened at this time. The 
images received through the parallactic lens 32 and the 
parallactic shutter 34 forms a parallactic image. Thus, the 
30 parallactic CCD 36 receives the parallactic image of the subject 
formed by the parallactic lens 32 and converts it to electronic 
signals . 

The normal image data input unit 24 inputs a normal image 
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photographed from a single viewpoint . The normal image data input 
unit 24 has a lens 25, a lens stop 26, a shutter 27, a color filter 
28 and a charge coupled device (CCD) 29. The lens 25 forms an 
image of a subject. The lens stop 2 6 adjusts aperture condition. 
5 The shutter 27 adjusts exposure time. The color filter 28 

separates RGB components of the light received through the lens 
25. The CCD 2 9 receives the image of the subject formed by the 
lens 25 and converts it to electric signals. 

10 The A/D converter 30 receives analog signals from the 

parallactic image data input unit 22 and the normal image data 
£=l input unit 24. The A/D converter 30 converts the received analog 

signals to digital signals and outputs the digital signals to the 
Ey memory 40. The memory 40 stores the input digital signals. It 

15 means that the memory 4 0 stores the data of the parallactic image, 
O the subject photographed by the parallactic image data input unit 

s 22, and the data of the normal image of the subject photographed 

rrf by the normal image data input unit 24 . 

J=S 20 The control unit 50 selects a best image from among a 

S plurality of normal images stored in the memory 40, based on a 

predetermined selection condition. The control unit 50 then 
outputs the data of the selected image. The control unit 50 
controls at least one of the following operations: focus 
25 condition of the lens 25, aperture condition of the lens stop 26, 
exposure time of the shutter 27, output signal of the CCD 29, 
condition of the parallactic shutter 34, and the output signal 
of the parallactic CCD 36. The control unit 50 controls the input 
unit 20 to continuously photograph the plurality of images. 

30 

The recording unit 90 records the image output from the 
control unit 50. The recording unit 90 may be, for example, a 
magnetic recording medium such as a floppy disk, or a nonvolatile 
memory such as flash memory. The output unit 92 outputs the image 
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recorded on the recording unit 90. The output unit 92 may be, 
for example, a printer or a monitor. The output unit 92 may be 
a small liquid crystal display (LCD) of the camera 10. In this 
case, the user can see the image selected by the control unit 50 
5 immediately after photographing the plurality of images. 

Fig. 2 is a block diagram of the control unit 50 according 
to the first embodiment. The control unit 50 includes a 
photographing control unit 55, an extractor 60, a 
10 condition-storing unit 70 and a selecting unit 80. 



^ The photographing control unit 55 controls at least one of 

'J3 the following: focus condition of the lens 25, focus condition 

fg of the parallactic lens 32, aperture condition of the lens stop 

yj 15 26, exposure time of the shutter 27, output signal of the CCD 29, 

y s 

Q condition of the parallactic shutter 34, and output signal of the 

I parallactic CCD 36. The photographing control unit 55 controls 

E3 the input unit 20, shown in Fig. 1, to continuously photograph 

p a plurality of images. 

K 20 

Q The extractor 60 receives the data of the normal images from 

the memory 40. The extractor 60 extracts an aimed object from 
each of the normal images based on the information obtained from 
the parallactic images and the normal images. The information 

25 includes image information of the normal image and depth 

information of the parallactic image. The aimed object defined 
here is an independent object at which a photographer aims when 
photographing. The aimed object may be, for example, a person 
in a room when the person and the objects in the room are 

30 photographed, a fish in an aquarium when the fish and the aquarium 
are photographed, or a bird stopping on a branch of a tree when 
the bird and the tree are photographed. 

The extractor 60 then detects a judgement location from the 
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aimed object based on the information obtained from the 
parallactic images and the normal images. The judgement location 
defined here is a location to which specific attention is paid 
when selecting the desirable image. The judgement location may 
5 be, for example, an eye of a person when the person is photographed, 
or a wing of a bird when the bird is photographed. The aimed object 
may be an area including the judgement location, extracted for 
a certain purpose. 

10 The condition-storing unit 70 stores predetermined 

selection conditions for selecting a best image, which will be 

^ referred to as a desired image, from among the plurality of normal 

y3 images. The condition-storing unit 70 includes a 

gg condition-setting unit, not shown in the drawings, by which a user 

H; 15 can select at least one of the selection conditions from among 

Q a plurality of selection conditions. 

S The selecting unit 80 selects a desired image which best 

m 

p satisfies the selection conditions stored in the 

U? 20 condition-storing unit 70 and set by the condition- sett ing unit 

Q 71. 

Since cameras are usually used to photograph human beings, 
a desired image means that the photographed person has a good 
25 appearance. The good appearance of the person may be when, for 
example, "the person is not blinking" , "the person' s eyes are not 
red-eyed", "the person is looking at the camera", or "the person 
is smiling" . 

30 In this embodiment, the method of selecting a desired image 

when a person is photographed will be explained. The 
condition-storing unit 70 stores conditions such as "the person 
is not blinking", "the person's eyes are not red-eyed", "the 
person is looking at the camera" , and "the person is smiling" as 
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the selection conditions. These selection conditions relate to 
the face of the person, and more specifically to eyes or mouth 
of the person. Therefore, it is assumed in this embodiment that 
the aimed object is the face area of the person and the judgement 
5 location is an eye or the mouth of the person. 

Each of the selection conditions has a reference situation 
for the judgement location, which should meet the requirements 
of the selection condition. The condition-storing unit 70 also 
10 stores the reference situations for the judgement location, each 
respectively corresponding to each of the selection conditions. 
_ The reference situations for the judgement location 

CI corresponding to each of the selection conditions will be 

03 described in the following. 

E 15 

Q For the conditions such as "the person is not blinking", 

' "the person's eyes are not red-eyed" and "the person is looking 

D at the camera" , the reference situation may relate to the shape 

oi 

q of the eye, color of the eye, and size of the eye. For the 

if- 20 condition such as "the person is smiling" , the reference 
Q situation may also relate to the size of the eye, as well as shape 

of the mouth, and size of the mouth. Whether each of the judgement 
locations satisfy each of these reference conditions or not is 
judged in accordance with predetermined algorithms based on 
25 experience. 



When the selection condition "the person is not blinking" 
is selected, the judgement location may be the eye of the person. 
The reference situation for the eye in this selection condition 
30 will be determined as follows. When a person blinks, his/her 
eyelid hides his/her eyeball. While he/she is blinking and 
his/her eye is partially closed, a white part of his/her eyeball 
is especially hidden by his/her eyelid. It means that when the 
person is not blinking, the white part of his/her eyeball should 
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be relatively large. Therefore, the reference situation for the 
selection condition "the person is not blinking" becomes "the 
white part of his/her eyeball has a large dimension" . 



red-eyed" is selected, the judgement location may be the eyes of 
the person. The reference situation for the eyes in this 
selection condition will be determined as follows. Eyes of a 
person are usually red-eyed when the person is photographed using 
10 a flash in a dark situation. This happens because the person's 
eyes cannot sensibly compensate for the sudden brightness and 
his/her pupils become red. It means that when the person's eyes 



look red-eyed, his/her pupils in each iris become red and the rest 
of the iris does not become red. Typically, people of Asian 



Em 15 descent have brown or dark brown colored irises, and people of 



European descent have green or blue colored irises. Therefore, 
the reference situation for the selection condition "the person' s 



eyes are not red-eyed" becomes "the red part in his/her iris has 
a small dimension" . 



When the selection condition "the person is looking at the 
camera" is selected, the judgement location may be the eye of the 
person. The reference situation for the eye in this selection 
condition will be determined as follows. When a person is looking 
25 at the camera, a line between the camera and the iris of the person 
and a normal vector of his/her iris are almost the same. Therefore, 
the reference situation for the selection condition "the person 
is looking at the camera" becomes "the normal vector of the iris 
in his/her eye is approximately equal to the angle of the line 
30 between the camera and his/her iris" . 

When the selection condition "the person is smiling" is 
selected, the judgement location may be the eyes and the mouth 
of the person. The reference situation for the eyes and the mouth 



5 



When the selection condition 



"the person's eyes are not 
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in this selection condition will be determined as follows. When 
a person is smiling, although it depends on each person, his/her 
eyes become relatively thin. At this time, although it depends 
on each person, his/her mouth expands right-and-left wards and 
his/her teeth are shown. Therefore, the reference situations for 
the selection condition "the person is smiling" become "the white 
part in his/her eyes has a small dimension" , "the width of his/her 
mouth is wide" and "the white area in his/her mouth has a large 
dimension" . 

The method of selecting a desired image from among the 
^ plurality of images will be explained in detail with reference 

yp to Figs . 3 and 4 . 

2 15 Fig. 3 is a functional block diagram of the extractor 60. 

q The extractor 60 includes a depth information extractor 62, an 

f~ image information extractor 64, an aimed object extractor 66 and 

O a judgement location detector 68. 

01 

Jjf 20 The depth information extractor 62 extracts the depth 

U 

q information indicating the distance to each of components of the 

subject, based on the data of the parallactic image received from 
the memory 40. It means that the depth information extractor 62 
determines a corresponding point for each of the components based 

25 on the parallactic image and gives a parallax amount. The depth 
information extractor 62 extracts the depth information based on 
the parallax amount of each of the components. Determining the 
corresponding point is a known technique, thus the explanation 
of this technique will be omitted. Extracting the depth 

30 information based on the parallax amount is also a known technique 
using the principle of triangulation, thus the explanation of 
this technique will be omitted. 

The image information extractor extracts the image 
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information for normal images, from the data for the normal images 
received from the memory 40. The image information includes, for 
example, data of the normal image such as luminescence 
distribution, intensity distribution, color distribution, 
5 texture distribution, and motion distribution. 

The aimed object extractor 66 extracts data for the face 
area of the person as the aimed object, based on the depth 
information and the image information. Each of the images may 
10 include, for example, a plural ity of components . The aimed obj ect 
extractor 66 recognizes each of the components based on the depth 
information. The aimed object extractor 66 then specifies the 

O 

face area by referring to the depth information and the image 
Zt information of each of the components. The method of specifying 

Cm 15 the face area will be described in the following. 

^ The aimed object extractor 66 receives the selection 

p condition from the condition- storing unit 70. The aimed object 

fn 

m extractor 66 extracts the aimed object based on the selection 

SjU 20 condition. In this embodiment, the aimed object is the face of 
□ the photographed person. Therefore, at first, the component 

including the face is specified depending on assumptions such as 
"the person should be close to the camera" , "the person should 
be in the middle of the image" , or "the proportional relationship 
25 of the height of the person to the width and height of the image 
should be within a predetermined range" . The distance from the 
camera to each of the components in the image is evaluated based 
on the depth information. The distance from the center of the 
image to each of the components in the image, and the proportional 
30 relationship of the height of the components are evaluated based 
on the image information. Each of the values is multiplied by 
predetermined constants corresponding to each condition. The 
multiplied values are added for each of the components. The added 
values are defined as weighted averages. The component having 
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a largest weighted average is extracted as the component 
including the aimed object. 

The constants by which the values for each of the components 
5 are multiplied may be predetermined based on the aimed object. 
In this embodiment, for example, the aimed object is assumed to 
be the face of the photographed person. Therefore, the aimed 
object extractor 66 specifies the area having a skin color as the 
face part, based on the image information. The colors of each 
10 of the components are evaluated based on the color distribution 
of the images. The values of the color distribution may also be 
multiplied by predetermined constants and the multiplied values 
are added for each of the components to give the weighted 
JfJ averages . 

m is 

5,5 f 

f=! The judgement location detector 68 detects the judgement 

■kssit 

^ location from the data for the face area extracted by the aimed 

O object extractor 66. The judgement location detector 68 receives 

fp 

the selection condition from the condition-storinq unit 70. The 

LJ 

fy 20 judgement location detector 68 detects the judgement location 

«3ST. 

q based on the selection condition. In this embodiment, the 

judgement location is an eye or the mouth of the photographed 
person. Therefore, the judgement location detector 68 detects 
the eyes and mouth from the face area. There is relatively little 

25 variation in eyes of people with respect to color, shapes or their 
place on the face. Therefore, patterns of eyes such as the color 
of the eyes, shape of the eyes, and the place of the eyes on the 
face are previously determined, and the parts which are 
approximately similar to the determined patterns of the eyes are 

30 recognized as the judgement location in the face. Similarly, 
there is relatively little variation in mouths of people with 
respect to color, shapes or place on the face. Therefore, 
patterns of the mouth are also previously determined and the parts 
which are approximately similar to the determined patterns of the 
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mouth are recognized as the judgement location in the face. 

The judgement location detector 68 then outputs the data 
for the detected judgement locations to the selecting unit 80. 

Fig. 4 is a functional block diagram of the selecting unit 
80. The selecting unit 80 includes a 
desired-aimed-object-selecting unit 82, a 

desired-image-selecting unit 84 and an image -composite unit 86. 
The desired-aimed-object-selecting unit 82 receives the data of 
the detected judgement locations from the extractor 60. The 
desired-aimed-object-selecting unit 82 receives the selection 
condition from the condition- storing unit 70. The 
desired-aimed-object-selecting unit 82 compares each of the 
judgement locations based on the reference situation for the 
selection condition. The desired aimed-obj ect - selecting unit 82 
then selects the judgement location which best satisfies the 
reference situation for the selection condition. 

When the selection condition "the person is not blinking" 
is selected, the judgement location is the eyes and the reference 
situation is "the white part of his/her eyeball has a large 
dimension", as described above. Therefore, the 

desired-aimed-object-selecting unit 82 calculates the dimension 
of the white part of the eye detected by the judgement location 
detector 68 for each of the images, based on the image information. 
The desired-aimed-object-selecting unit 82 selects the eye 
having the largest dimension of the white part as the desired 
judgement location. People usually blink both eyes at a same time, 
therefore, the desired-aimed-object-selecting unit 82 may check 
only one of the eyes of the photographed person. However, by 
checking both eyes, the desired judgement location can be 
selected more precisely. 
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When the selection condition "the person's eyes are not 
red-eyed" is selected, the judgement location is the eyes and the 
reference situation is "the red part in his/her iris has a small 
dimension", as described above. Therefore, the 
5 desired-aimed-obj ect - select ing unit 82 calculates the dimension 
of the red part in the iris of the eye detected by the judgement 
location detector 68 for each of the images, based on the image 
information. The iris of his/her eye is recognized as being a 
cylindrical or elliptic area whose circumference has a brown or 
10 dark brown color. The desired-aimed-object-selecting unit 82 
selects the eye having the smallest dimension of the red part as 

_ the desired judgement location. Both eyes of people are usually 

£3 

yQ red eyed at the same time, therefore, the 

24 desired-aimed-object-selecting unit 82 may check only one of the 
GO 15 eyes of the photographed person. However, by checking both of 

q his/her eyes, the desired judgement location can be selected more 

^ precisely. 

s 

f=J 

'%! When the selection condition "the person is looking at the 

20 camera" is selected, the judgement location is the eye and the 

Q reference situation is "the normal vector of the iris in his/her 

eye is approximately equal to the angle of the line between the 
camera and his/her iris", as described above. Therefore, the 
desired-aimed-object-selecting unit 82 recognizes the iris as 
25 being a cylindrical or elliptic area whose circumference has a 
brown or dark brown color. The desired-aimed-object-selecting 
unit 82 then recognizes the center of the iris and the normal 
vector of the center of the iris. The 

desired-aimed-object-selecting unit 82 selects the eye having 
30 the normal vector of the iris which is closest to the line between 
the camera and the iris, as the desired judgement location. 

The normal vector of the iris can be obtained from the 
relative position of the camera and the face of the person, the 
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relative position of the face and the eyes of the person, and the 
relative position of the eyes and the irises of the person. The 
desired-aimed-obj ect-selecting unit 82 may select the desired 
judgement location based on the normal vector obtained from these 
5 relative positions. 

When the selection condition "the person is smiling" is 
selected, the judgement location is the eyes or the mouth and the 
reference situation is "the white part in his/her eye has a small 
10 dimension" , "the width of his/her mouth is wide" or "the white 
part in his/her mouth has a large dimension" , as described above. 
Therefore, the desired-aimed-obj ect-selecting unit 82 

% calculates the dimension of the white part of the eye, the width 

of the mouth, and the dimension of the white part of the mouth 

ffi. 15 detected by the judgement location detector 68 for each of the 

^ images, based on the image information. The 

desired-aimed-obj ect -selecting unit 82 selects the eye having 

p the smallest dimension of the white part, the mouth having the 

JL ? widest width, or the mouth having the largest dimension of the 

FLj 20 white part as the desired judgement location. 

o 

The desired-aimed-obj ect-selecting unit 82 selects the 
aimed object including the desired judgement location selected 
by the operation, as described above. 

25 

The desired-image-selecting unit 84 selects the desired 
image by specifying the image including the desired judgement 
location . 

30 As described above, the control unit 50 extracts the face 

of the photographed person as the aimed object and detects the 
judgement location from each of the aimed objects. The control 
unit 50 then selects a desired judgement location most satisfying 
the selection condition. Thus, the control unit 50 automatically 

FF-0006 

j 



• # 



selects a desired image including the desired judgement location 
from among the plurality of images, without bothering the 
photographer or the photographed person. 

5 The method of selecting a desired image when a plurality 

of people is photographed will be explained next. 

When each of the images includes a plurality of people, the 
extractor 60 extracts the aimed object and detects the judgement 
10 locations for each of the people. It means that the aimed object 
extractor 66 extracts the face parts for each of the people from 
each of the images. The judgement location extractor 68 detects 
4? the eyes or the mouth for each of the people from each of the 

Uj images. 

m 15 When each of the images includes a plurality of people, the 

if; desired-aimed-obj ect-selecting unit 82 selects the desired aimed 

!__= 

M objects each respectively including the desired judgement 

f*% locations selected for each of the people. The 

0 1 desired-image-selecting unit 84 specifies desired aimed images, 
m 20 each respectively including the desired aimed object for each of 

IS the people . 

The desired image may be different for each person when a 
plurality of people is photographed. Therefore, the 
25 image-composite unit 86 composes the desired aimed objects for 
each of the photographed people selected by the 
desired-aimed-obj ect-selecting unit 82 and forms a composite 
image. The image-composite unit 86 outputs the composite image 
to the recording unit 90. 

30 

The image -composite unit 86 normally does not composite the 
images when only one person is photographed. However, there may 
be a problem even when one person is photographed. For example, 
the image itself may not be desirable although the aimed object 
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included in the image is desirable, because the face part and the 
other subjects in the image have different tone levels. In such 
a case, the image-composite unit 86 composites the desired aimed 
object with a desirable image having a desirable background. 

5 

In addition, as the data of the face part of the person can 
be extracted from the data of the image, the face part and the 
rest of the image may be separately corrected and then may be 
composited, by the image-composite unit 86. For example, when 
10 compressing the image, the face part and the image may be 

separately compressed so that the face part can be less compressed 
_ and may be composited by the image -composite unit 86 . Furthermore, 

yg only the face part may be magnified and may be composed with the 

JfJ original background so that the face part can be emphasized. 

01 15 

fn 

f4 Fig. 5 is a flowchart showing the method of selecting a 

u- desired image from among the plurality of images photographed by 

Q the camera 10. The camera 10 starts photographing the subject 

JL S when a release button is pushed (S100) . When the camera 10 starts 

1=1 

fU 20 photographing, data of a parallactic image are input from the 
2 parallactic image data input unit (S102) . At the same time, data 

of a plurality of normal images are input from the normal image 
data input unit (S104) . The extractor 60 extracts the face part 
of the photographed person as the aimed object based on the depth 
25 information and the image information (S106) . The extractor 60 
then detects the judgement location based on the image 
information of the face part (S108) . The selecting unit 80 
selects a desired image from among the plurality of normal images 
including a desired aimed object whose judgement location 
30 satisfies the selection condition (S112) . 

The selection unit 80 then composites the selected desired 
images if necessary (S114). 
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The desired image is output to the recording unit 90 . The 
recording unit 9 0 records the received desired image (S116) . The 
desired image recorded on the recording unit is output from the 
output unit 92 . 

5 

Fig. 6 is a flowchart showing in detail the method of 
extracting the face part as the aimed object (S106) of Fig. 5. 
The depth information is extracted based on the parallactic image 
(S130) , and the image information is extracted based on each of 
10 the normal images (S132) . The face part of the photographed 
person is extracted from each of the images based on the depth 
information and the image information (S134) . When each of the 
[fj images includes a plurality of people, the 

!£j aimed-object-extracting unit 66 extracts the face parts for all 

m 15 of the people from each of the images (S136) . This extracting 
~ method is repeatedly done for each of the images (S138) . When 

H the data of the face parts are extracted from all of the images, 

p the process moves to the next step S108 . 

fU 20 Fig. 7 is a flowchart showing in detail the method of 

gj detecting the judgement location (S108) of Fig. 5. The judgement 

location is detected based on the image information of each of 
the faces (S140) . When each of the images includes a plurality 
of people, the j udgement - location-detect ing unit 68 detects the 
25 judgement locations for all of the people from each of the images 
(S142) . This detecting method is repeatedly done for each of the 
images (S144) . When the data for the judgement locations are 
detected from all of the images, the process moves to the next 
step S112 . 

30 

Fig. 8 is a flowchart showing in detail the method of 
selecting the desired image (S112) of Fig. 5. The data of the 
judgement location detected in the step 108 is compared for each 
of the images, based on the reference situation corresponding to 
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each of the selection conditions. The 

desired-aimed-obj ect-selecting unit 82 selects the judgement 
location most satisfying the selection condition, as the desired 
aimed object (S150) . The desired-aimed-obj ect-selecting unit 82 
5 then selects the aimed object including the desired judgement 
location, as the desired aimed object (S152) . The 
desired-image-selecting unit 84 specifies the image including 
the desired aimed object as the desired image (S154) . When each 
of the images includes a plurality of people, the desired images 
10 are selected for all of the people (S156) . 



Fig. 9 shows an image selecting apparatus of the second 
y embodiment according to the present invention. The image 

U] selecting apparatus in this embodiment is a laboratory system, 

jyi 15 The laboratory system of this embodiment includes a reading unit 
110, a memory 120 , a control unit 130, a recording unit 140 and 
m an output unit 150 . 

5 

0] The reading unit 110 reads data of a plurality of images 

fy 20 and information related to each of the images from a recording 
y medium for a digital camera, or a film. The reading unit 110 may 

be a floppy disk drive, for example, and reads the data of the 
images and the information from a floppy disk or a removable flash 
memory. The reading unit 110 may have a communication unit 
25 through which data of a plurality of images photographed by a 
digital camera and information related to each of the images are 
transmitted over a computer network such as the internet. Thus, 
the reading unit 110 can read the data of the images and the 
information thereof . 



30 



The reading unit 110 may be a film scanner, for example, 
and reads the data of the images and the information thereof from 
a film. There is provided in the film a space for the information 
related to the images. The reading unit 110 reads data of the 
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images and the information related to each of the images. 

The memory 12 0 stores the data of the plurality of images 
and the information thereof read by the reading unit 110 . In this 
embodiment, the memory 12 0 stores depth information and image 
information as the information related to the images. 

The control unit 13 0 has a same function as the control unit 
50 of the camera 10 according to the first embodiment. The control 
unit 130 selects a desired image most satisfying the selection 
conditions from among a plurality of images stored in the memory 
120. The control unit 130 extracts a face part of a person in 
the image based on the depth information and the image information 
stored in the memory 120. The control unit 130 then detects the 
judgement location based on the image information of the 
extracted face part. 

As for the films, the space for information related to the 
images is not large, therefore, only the image information 
related to the face part of the person may be written. In this 
case, the judgement location is detected based on the image 
information of the face part written of the film. As for the 
recording medium of the digital camera, parallactic images may 
be stored in the recording medium, and the depth information may 
be extracted based on the parallactic images. 

The control unit 130 selects a desired aimed object 
satisfying the selection condition, based on the detected 
judgement location. The control unit 130 then selects a desired 
image from among the plurality of images. The desired image 
selected by the control unit 130 is recorded on the recording unit 
140. The desired image recorded on the recording unit 140 is 
output from the output unit 150. The output unit 150 may be a 
color laser printer, or a digital printer. 
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The image selecting apparatus may be a workstation, or a 
system using a computer such as a personal computer. The computer 
serves as the memory 12 0, the control unit 13 0 and the recording 
unit 14 0 shown in Fig. 9. Peripheral apparatus that together 
compose the system with the computer, may serve as the reading 
unit 110 and the output unit 150 shown in Fig. 9. 

As for the image selecting apparatus, the method of 
selecting a desired image from among a plurality of images may 
be provided with a software program stored in a recording medium 
2 00 shown in Fig . 9. In this case, the recording medium 200 stores 
therein a program executed by a computer, to perform a method 
comprising extracting data of an aimed object from each of the 
plurality of images, and selecting a desired image including a 
desired aimed object from among the plurality of images, the 
desired aimed object satisfying a predetermined selection 
condition for a desirable aimed object. 

As described above, the control unit 13 0 extracts the face 
part of the photographed person as the aimed object and detects 
the judgement location from each of the aimed objects. The 
control unit 13 0 then selects a desired aimed object including 
the judgement location most satisfying the selection condition, 
based on the detected judgement location. Thus, the control unit 
130 automatically selects a desired image including the desired 
aimed object from among the plurality of images, without 
bothering the photographer or the photographed person. 

Furthermore, even when a lot of people are photographed in 
an image, the face parts are extracted for each of the people. 
Then, the desired face part including the judgement location most 
satisfying the selection condition is selected for each of the 
people. Therefore, by compositing the desired aimed objects for 
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each of the people, a desired image can be automatically obtained. 

As described above, a desired image satisfying a 
predetermined selection condition can be selected from among a 
plurality of images according to the present invention. 

Although the present invention has been described by way 
of exemplary embodiments, it should be understood that many 
changes and substitutions may be made by those skilled in the art 
without departing from the spirit and the scope of the present 
invention which is defined only by the appended claims. 
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